aes23 International Evaluation Conference

Gerard Atkinson (ARTD Consultants, -), Michael Brooks (ARTD Consultants, AU)

Machine learning is the hot topic of 2023, and it seems that every day brings a new application. Beyond the questions of whether the chatbots are going to take our jobs, are fundamental questions around whether machine learning actually works better than humans at certain tasks. For evaluation, one of these tasks is topic classification, a core part of the qualitative researcher's toolkit.

In this presentation we present the results of an experimental test of machine learning against fully human and hybrid approaches to topic classification. We take a range of approaches from chatbots through to automated processing and classify performance on a range of metrics, including speed, accuracy, and ease of use. The results of our experiment will give you guidance on how and when machine learning can be used to enhance one part of your evaluation practice.